Skip to content

[ci] Add GPG-signed DEB packages#5180

Open
PlatCore wants to merge 41 commits into
masterfrom
PlatCore/5109-add-deb-gpg-signing-v2
Open

[ci] Add GPG-signed DEB packages#5180
PlatCore wants to merge 41 commits into
masterfrom
PlatCore/5109-add-deb-gpg-signing-v2

Conversation

@PlatCore
Copy link
Copy Markdown
Contributor

@PlatCore PlatCore commented Apr 6, 2026

Why this should be merged

DEB packages are currently unsigned and built with an ad-hoc dpkg-deb flow. This adds GPG-signed DEB packages for avalanchego and subnet-evm, modelled on the existing RPM pipeline — containerized build, nfpm-native signing, end-to-end validation on both supported Ubuntu releases.

Depends on #5179.

How this works

Mirrors the RPM pipeline pattern with a DEB-specific container:

  • Dockerfile.deb — Ubuntu 22.04 builder image (gcc, gnupg, Go, nfpm).
  • build-package.sh + lib-build-common.sh — unified RPM/DEB build helper. Builds the binary, sets up GPG (ephemeral key for PR/local; secrets.RPM_GPG_PRIVATE_KEY for releases), and packages with nfpm.
  • nfpm DEB templatesdeb.signature.key_file: "${NFPM_SIGNING_KEY}" so nfpm signs in-process (no post-build signing step). Install paths: /usr/local/bin/avalanchego and /usr/local/lib/avalanchego/plugins/<VM_ID>.
  • validate-deb.sh — in both ubuntu:22.04 and ubuntu:24.04: extracts _gpgorigin from the ar archive, runs gpg --verify against debian-binary + control.tar.* + data.tar.*, then installs the packages and runs the shared smoke test.
  • Taskfile.yml — Docker-run-based DEB tasks matching the RPM task pattern. task packaging:test-build-debs works locally on macOS.
  • build-deb-release.yml — invokes task packaging:test-build-debs. Uploads artifacts on non-PR runs and pushes to S3 on tag push / workflow_dispatch. workflow-setup-packaging.sh fails fast if the signing key secret is absent on a release event.
  • Removes: build-ubuntu-amd64-release.yml, build-ubuntu-arm64-release.yml, build-deb-pkg.sh, debian/template/control.

Key design decisions:

  • nfpm-native signing (_gpgorigin ar member) verified with gpg --verify. Works identically on jammy and noble — no dpkg-sig anywhere.
  • Shared GPG key with RPM via secrets.RPM_GPG_PRIVATE_KEY; the passphrase is forwarded into the container with a value-less -e VAR flag so secrets with whitespace or shell metacharacters reach nfpm intact.

How this was tested

  • task packaging:test-build-debs locally on macOS (Docker): both packages built and signed by nfpm, gpg --verify good in jammy and noble containers, install + smoke test passed on both.
  • Targeted tests for the workflow-setup signing-key gate, the overlay replace, the VM-ID resolver fallback, and task --dry of the build task confirming the passphrase is no longer interpolated into the docker command-line.
  • CI: workflow triggers on PRs touching .github/packaging/**.

Need to be documented in RELEASES.md?

No

@PlatCore PlatCore requested a review from a team as a code owner April 6, 2026 03:47
@PlatCore PlatCore requested a review from maru-ava April 6, 2026 04:23
@PlatCore PlatCore self-assigned this Apr 6, 2026
@PlatCore PlatCore added ci This focuses on changes to the CI process devinfra labels Apr 6, 2026
@PlatCore PlatCore linked an issue Apr 6, 2026 that may be closed by this pull request
1 task
Comment thread .github/packaging/scripts/build-deb.sh Outdated
Comment thread .github/packaging/nfpm/avalanchego.yml Outdated
Comment thread .github/packaging/Dockerfile
Comment thread .github/workflows/build-deb-release.yml Outdated
Comment thread .github/packaging/scripts/validate-deb.sh
@PlatCore PlatCore force-pushed the PlatCore/5109-add-deb-gpg-signing-v2 branch 3 times, most recently from c6cdc18 to 84aa1f4 Compare April 7, 2026 03:31
@PlatCore PlatCore requested a review from maru-ava April 8, 2026 02:18
Copy link
Copy Markdown
Contributor

@maru-ava maru-ava left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm intentionally keeping this review narrow until 5179 is ready. That said, there is at least one DEB-specific correctness issue here that seems worth calling out now: the release-signing path appears to rely on passphrase handling that PR CI does not exercise, and the current gpg-preset-passphrase invocation does not look correct.

Comment thread .github/packaging/scripts/lib-build-deb.sh Outdated
@PlatCore PlatCore force-pushed the PlatCore/5109-refactor-rpm-for-reuse-v2 branch from f2c029c to 402dd0c Compare April 12, 2026 05:15
Comment thread .github/packaging/scripts/build-package.sh Outdated
@PlatCore PlatCore moved this to In Progress 🏗️ in avalanchego Apr 13, 2026
@PlatCore PlatCore force-pushed the PlatCore/5109-refactor-rpm-for-reuse-v2 branch from 402dd0c to d336cf3 Compare April 13, 2026 21:53
@PlatCore PlatCore force-pushed the PlatCore/5109-add-deb-gpg-signing-v2 branch 2 times, most recently from 5f4fb6a to e9e97d1 Compare April 13, 2026 23:05
@PlatCore PlatCore requested a review from maru-ava April 14, 2026 01:10
@PlatCore PlatCore moved this from In Progress 🏗️ to Ready 🚦 in avalanchego Apr 14, 2026
@PlatCore PlatCore force-pushed the PlatCore/5109-add-deb-gpg-signing-v2 branch from be87574 to e03431d Compare April 16, 2026 16:47
@PlatCore PlatCore linked an issue Apr 27, 2026 that may be closed by this pull request
@PlatCore
Copy link
Copy Markdown
Contributor Author

  • Documented the .deb packaging process.
    • Linked shared details to the RPM packaging doc.

PlatCore added 30 commits May 27, 2026 16:37
The setup-packaging Taskfile task was removed and replaced by
workflow-setup-packaging.sh. The RPM workflow was updated but
the DEB workflow still referenced the old task name.
RPM refactor. After rebasing this branch onto the post-excision tip
of #5179, the hooks need to return as commits owned by #5180 so
they survive the eventual merge into master.

build-package.sh: source lib-build-${fmt_lower}.sh on FORMAT=DEB,
add the DEB arm to the GPG_KEY_FILE case, configure gpg-agent via
setup_deb_gpg_agent before setup_gpg, cache the passphrase via
cache_deb_gpg_passphrase for non-interactive dpkg-sig signing, and
invoke sign_deb_package post-nfpm (nfpm's openpgp signatures are
incompatible with dpkg-sig --verify).

lib-validate-common.sh: re-add the DEB arm of detect_host_arch's
format dispatch (x86_64 -> amd64, aarch64/arm64 -> arm64).

The corresponding helper functions are defined in lib-build-deb.sh.

Verified locally: task -t .github/packaging/Taskfile.yml
test-build-debs passes end-to-end. Both .debs built, dpkg-sig
signatures verify (GOODSIG), and install + smoke test pass in
fresh ubuntu:22.04 (jammy) and ubuntu:24.04 (noble) containers.
Mirrors the RPM template refactor. build-package.sh exports only
${BINARY_PATH}; the DEB templates' references to
${AVALANCHEGO_BINARY} / ${SUBNET_EVM_BINARY} resolved to empty via
envsubst, causing nfpm to fail with 'glob failed: no matching files'
on the amd64 CI run.
actionlint flagged 'custom-arm64-jammy' as an unknown label;
master migrated arm64 to the GitHub-hosted ubuntu-22.04-arm
runner. The DEB workflow was authored before that sweep.
nfpm now signs DEBs inline via deb.signature.key_file, matching the RPM
side. Validation extracts the _gpgorigin ar member and runs gpg --verify
against debian-binary+control+data, so the same flow works on jammy and
noble (dpkg-sig was unavailable in noble). Builder image, build helper,
Taskfile comment, and design doc no longer reference dpkg-sig.
- workflow-setup-packaging.sh now fails fast when no signing key is
  provided for a tag push or workflow_dispatch (previously fell back
  to ephemeral-key generation, so unsigned-by-real-key packages could
  reach S3).
- build-{deb,rpm}-release.yml replace .github/packaging atomically
  (rm -rf + mv) instead of `cp -r` into an existing directory, which
  was leaving the checked-out tag's stale Taskfile in place and
  copying the overlay to .github/packaging/packaging/.
- resolve_subnet_evm_vm_id falls back to grepping
  graft/subnet-evm/scripts/constants.sh when the new
  default-vm-data.sh is absent, restoring workflow_dispatch builds of
  tags predating the data file.
- Taskfile.yml forwards NFPM_{RPM,DEB}_PASSPHRASE via a task-level
  `env:` block and value-less `-e NFPM_*_PASSPHRASE` so passphrases
  containing whitespace or shell metacharacters reach the container
  intact (no shell evaluation of the secret).
Taskfile.yml: the four build-{avalanchego,subnet-evm}-{rpm,deb} tasks
collapse into one internal `build-package` task that the four wrappers
delegate to with `task: build-package` + vars. The wrappers only carry
the format/output/builder-image differences; the docker-run block lives
in one place.

build-package.sh: switch the format env var from PKG_FORMAT (RPM|DEB) to
NFPM_PACKAGER (rpm|deb) to match nfpm's CLI naming, and collapse
RPM_GPG_KEY_FILE / DEB_GPG_KEY_FILE into a single GPG_KEY_FILE since the
caller already picks the right one.

The value-less `-e NFPM_*_PASSPHRASE` passphrase forwarding survives the
refactor (verified via task --dry with a metachar-laden passphrase).
Moves VERSION (derived from TAG via trimPrefix "v") and TAG into the
build-package task's env: block and switches their docker flags to the
value-less `-e VAR` form, matching the pattern already used for the
NFPM_*_PASSPHRASE secrets. Keeps the docker invocation free of any
template-substituted values that could carry shell metacharacters.
Aligns the RPM-side var names with the DEB-side pattern (format prefix
between PACKAGING_ and the rest):

  PACKAGING_HOST_ARCH        -> PACKAGING_RPM_HOST_ARCH
  PACKAGING_HOST_DEB_ARCH    -> PACKAGING_DEB_HOST_ARCH
  PACKAGING_DOCKER_IMAGE     -> PACKAGING_RPM_DOCKER_IMAGE
  PACKAGING_OUTPUT_DIR       -> PACKAGING_RPM_OUTPUT_DIR

All references inside the Taskfile updated; no live consumers outside it.
Replaces docs/design/deb-packaging.md and docs/design/rpm-packaging.md
with a single shared design doc per reviewer comment on PR #5180. The
new doc has format-specific subsections for RPM and DEB and shared
subsections for build tooling, GPG signing, version smoke test, CI
workflow, and architecture mapping.

Also brings the prose in line with the current code:

- VM-ID source notes default-vm-data.sh with a fallback to constants.sh
  for older tags, matching the resolver in lib-build-common.sh.
- CI workflow examples use the actual --taskfile invocation form used
  in build-{rpm,deb}-release.yml, with a note about the packaging:
  namespace available locally via the root Taskfile include.
- GPG-signing section documents the fail-fast release-key gate that
  workflow-setup-packaging.sh enforces for tag pushes and
  workflow_dispatch.
The `.github/actions/setup-packaging/**` composite-action path was
removed from this branch, so the pull_request.paths filter referencing
it never matches. Drop it from both build-rpm-release.yml and
build-deb-release.yml.
Both release workflows pipe the same `gpg-key-file` step output into
the build env, and the per-format wrappers immediately funnel that into
build-package's GPG_KEY_FILE var. The {RPM,DEB}_ prefix carried no
information.

- Taskfile: four wrappers now read .GPG_KEY_FILE directly instead of
  the format-prefixed equivalents.
- build-deb-release.yml: env block sets GPG_KEY_FILE (was
  DEB_GPG_KEY_FILE) to match the already-normalized RPM workflow.
- build-deb-release.yml: GPG_KEY_PASSPHRASE replaces NFPM_DEB_PASSPHRASE
  (the latter was no longer read by the Taskfile; releases were silently
  unsigned) and is gated on non-pull_request like the RPM side.

NFPM_RPM_PASSPHRASE / NFPM_DEB_PASSPHRASE remain *inside*
build-package.sh as the nfpm-mandated env var names; the script
re-exports GPG_KEY_PASSPHRASE under the right NFPM_<FORMAT>_PASSPHRASE
just before invoking nfpm.
The RPM builder-image task was the only build-* task that omitted the
format prefix. With a parallel build-deb-builder-docker-image already in
place, the RPM task is now build-rpm-builder-docker-image so every
build-* task carries the same format suffix (or is the format-agnostic
internal build-package).

- Task definition at line 59.
- deps: refs in build-avalanchego-rpm and build-subnet-evm-rpm.

The underlying scripts/build-builder-image.sh stays unrenamed — it is
already format-agnostic (drives off the DOCKERFILE arg).
build-package.sh sets set -euo pipefail. When the Taskfile invokes
build-package without a real signing key, the docker run omits the
`-e GPG_KEY_FILE` flag entirely (Task conditional), so the container env
is unset. The script then references ${GPG_KEY_FILE} in the ephemeral-
key branch and aborts on `unbound variable` before signing setup runs.

Default GPG_KEY_FILE to empty alongside the existing NFPM_*_PASSPHRASE
defaults; the ephemeral-key conditional handles the empty case correctly.
f946cc6 renamed PACKAGING_HOST_ARCH to PACKAGING_RPM_HOST_ARCH but
missed the default chain in the validate-rpms task. With no explicit
PACKAGE_ARCH (the normal CI / test-build-rpms path), PACKAGE_ARCH
resolves to empty and validate-rpm.sh aborts at its
`: "${PACKAGE_ARCH:?...}"` assertion -- after the packages have been
built.
build-package.sh now exports the public key as
${pkg_format_upper}-GPG-KEY-avalanchego, so the RPM build produces
build/rpm/RPM-GPG-KEY-avalanchego. The two consumers still referenced
the unprefixed path:

- validate-rpm.sh checked /rpms/GPG-KEY-avalanchego and silently skipped
  signature verification on signed RPMs.
- build-rpm-release.yml uploaded build/rpm/GPG-KEY-avalanchego, so the
  release artifact never included the key.

Mirror the DEB side (which already uses DEB-GPG-KEY-avalanchego
end-to-end).
build-deb runs on pull_request and executes PR-controlled packaging
scripts. Granting id-token: write at the job level let PR scripts
request a GitHub OIDC JWT regardless of which steps actually ran.

Split into two jobs:

- build-deb: contents: read only. Builds, validates, and uploads
  artifacts on non-PR runs. Exposes the resolved tag as a job output.
- upload-debs-s3: needs: build-deb; runs only on tag push or
  workflow_dispatch; has id-token: write. Downloads artifacts and
  pushes them to S3.

The PR-executed job can no longer mint OIDC tokens; release uploads
keep working via the new dedicated job.
resolve_subnet_evm_vm_id sources constants.sh inside SUBNET_EVM_VM_ID=$(...)
so it captures whatever stdout the sourced file produces along with the
trailing echo "${DEFAULT_VM_ID}".

Older subnet-evm constants.sh revisions (e.g., v1.14.1 line 62) do
`echo "Using branch: ${CURRENT_BRANCH}"` at source time. Under
workflow_dispatch tag-overlay builds this contaminates SUBNET_EVM_VM_ID
with a "Using branch: ..." line, which then corrupts the rendered nfpm
yaml plugin path.

Redirect the source's stdout to /dev/null; the explicit
`echo "${DEFAULT_VM_ID}"` at the end of the subshell is the only thing
captured.
PR #5179 (Thread #7, pushed as 7245fa9) shrank smoke-test.sh from
4 args to 3: the caller now passes the composed subnet-evm plugin
binary path instead of the plugin dir + VM ID. validate-rpm.sh was
updated in the same commit; validate-deb.sh was missed and kept the
old 4-arg invocation.

After rebasing onto PR #5179's tip the contract drift surfaced in
build-deb-packages CI: positional-arg slip made $2 the plugins
*directory*, and smoke-test.sh tried to exec it
("Is a directory", exit 126).

Mirror the validate-rpm.sh fix.
Four regressions from PR #5179's review outcomes were carried into
the DEB-side code on PR #5180:

R1 — build-package.sh: remove the "Required/Optional env vars"
header docstring reintroduced by the NFPM_PACKAGER refactor.
The ${VAR:?msg} asserts already carry inline role descriptions
(PR #5179 Threads #9/#10).

R2 — validate-deb.sh: same header-docstring removal. Add
PACKAGE_ARCH as a required assert with role description, mirroring
validate-rpm.sh (PR #5179 Threads #10/#11).

R3 — validate-deb.sh: convert the "Skipping GPG verification
(unsigned build)" else branch to a fatal error. The pipeline
always produces a signed key (PR #5179 Thread #8).

R4 — Delete lib-validate-common.sh (reintroduced during the
PR #5180 rebase with detect_host_arch). The Taskfile already
computes host arch via PACKAGING_DEB_HOST_ARCH; validate-debs
now passes PACKAGE_ARCH through its env block, mirroring
validate-rpms (PR #5179 Thread #6).
Concrete fixes:
- subnet-evm-deb.yml: fix stale comment referencing non-existent
  default-vm-data.sh; the actual source is constants.sh (matches
  the RPM config and resolve_subnet_evm_vm_id in lib-build-common.sh).
- avalanchego-deb.yml + subnet-evm-deb.yml: bump libc6 dependency
  from >= 2.34 to >= 2.35 to match the DEB builder's Ubuntu 22.04
  glibc version (2.34 is the RPM/Rocky 9 floor).
- build-package.sh: remove redundant static exports of
  NFPM_RPM_PASSPHRASE and NFPM_DEB_PASSPHRASE; the dynamic export
  on line 49 already sets the correct format-specific var.
- Taskfile.yml:
  - Remove undefined .RPM_ARCH / .DEB_ARCH from PACKAGE_ARCH
    default chains (never set by any caller; dead chain links).
  - Move PACKAGING_DEB_HOST_ARCH adjacent to PACKAGING_RPM_HOST_ARCH
    for easy comparison.
  - Replace `sh: echo "${PACKAGING_TAG:-v0.0.0}"` with Go template
    `{{env "PACKAGING_TAG" | default "v0.0.0"}}` (no shell spawn).
  - Sort tasks alphabetically so RPM/DEB equivalents are
    side-by-side, per reviewer request for sorted ordering.
  - Remove unused PASSPHRASE_ENV var from build-package task.

Verified: test-build-debs and test-build-rpms both pass end-to-end.
Move the DEB + RPM design content from docs/design/linux-packaging.md
into the canonical .github/packaging/README.md per reviewer comment
and the convention established in #5373.

Fixed stale details while moving: constants.sh VM ID source and
libc6 (>= 2.35) DEB dependency floor.
Addresses the lightweight code-side feedback from the 2026-05-27 review
batch on #5180. Functional behavior unchanged; verified end-to-end via
`task -t .github/packaging/Taskfile.yml test-build-rpms` and the DEB
counterpart.

- build-package.sh: restore the `${PACKAGE:?…}` assert (was missing
  entirely) and enrich all asserts with inline role descriptions,
  matching the inline-assert convention from #5179 Thread #10. Drops
  the prefix from the exported public key (`GPG_PUBLIC_KEY` is now
  unprefixed — see below).
- lib-build-common.sh: drop the redundant `>/dev/null` and the
  misleading comment in `resolve_subnet_evm_vm_id`. The grouped block
  already routes its stdout to stderr via `>&2`, so the inner redirect
  was a no-op and the comment described a problem the existing
  redirect already handled.
- GPG-KEY filename: drop the format prefix end-to-end. RPM and DEB
  builds already separate outputs by directory (`build/rpm/` vs
  `build/deb/`, `/rpms/` vs `/debs/`, separate workflow artifact
  paths and S3 prefixes), so the per-format prefix was redundant.
  Updated build-package.sh, validate-rpm.sh, validate-deb.sh,
  build-rpm-release.yml (artifact upload include), build-deb-release.yml
  (artifact upload include + S3 `aws s3 cp` source), and the README.
- Dockerfile.{rpm,deb}: drop `=INVALID` from `GO_VERSION` and
  `GO_CHECKSUM` ARGs. Post-`FROM` ARGs don't benefit from the sentinel
  default — an unset value would behave identically (the
  `sha256sum -c` step catches the bad download in either case).
- Taskfile.yml: rename `build-{rpm,deb}-builder-docker-image` →
  `build-builder-docker-image-{rpm,deb}` so the task names sort
  consistently with the other format-suffixed tasks. Reordered the
  two task definitions to match the alphabetical order. Added an
  explicit `requires:` block to the internal `build-package` task
  surfacing its API contract (PACKAGE, NFPM_PACKAGER, TAG,
  PACKAGE_ARCH, OUTPUT_DIR, DOCKER_IMAGE).
- validate-deb.sh: drop the stale dpkg-sig advocacy sentence from the
  `_gpgorigin` verification comment. dpkg-sig isn't used; the comment
  now describes only what we actually do.
- build-deb-release.yml: add the top-level
  `FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true` env block that the RPM
  workflow and the deleted DEB workflows already set. Without it,
  Node-20 JS actions (`actions/checkout`, the setup-go composite,
  artifact actions, AWS credentials action) can fail during action
  setup in Node 24-gated environments before packaging starts.

Addresses
- https://github.com/ava-labs/avalanchego/pull/5180/changes#r3306495025
- https://github.com/ava-labs/avalanchego/pull/5180/changes#r3306509777
- https://github.com/ava-labs/avalanchego/pull/5180/changes#r3306532461
- https://github.com/ava-labs/avalanchego/pull/5180/changes#r3306542567
- https://github.com/ava-labs/avalanchego/pull/5180/changes#r3167909431
- https://github.com/ava-labs/avalanchego/pull/5180/changes#r3310777624
- https://github.com/ava-labs/avalanchego/pull/5180/changes#r3310801323
Per maru-ava feedback on thread #5180 (build-deb-release.yml:1 +
workflow-setup-packaging.sh:44): the post-overlay steps for RPM and
DEB are nearly identical, and the release-key gate was duplicated
between the workflow YAML and the setup script. Factor both into a
shared composite action so common behavior can't diverge.

- Add `.github/packaging/actions/build-package/action.yml` —
  composite action that handles setup-go, workflow-setup-packaging,
  build-and-validate (task test-build-${format}s), artifact upload
  (gated on release builds), and cleanup. Format/arch differences
  come in as inputs. Lives under .github/packaging/ so the
  workflow-branch overlay step (used for manual rebuilds of older
  tags) makes it available alongside the rest of the packaging tree.
  Approach validated by maru-ava's PoC in #5429.

- Add `.github/workflows/build-linux-packages.yml` — single workflow
  with a format × arch matrix delegating bulk work to the composite
  action. The DEB-only S3 upload remains a separate release-only job
  (id-token: write stays out of the build job). RPM S3 publishing
  can land as a sibling job later when the RPM-side S3 path is
  defined.

- Delete `.github/workflows/build-{rpm,deb}-release.yml` — both are
  subsumed by build-linux-packages.yml.

- workflow-setup-packaging.sh: drop the event-type sniffing block
  (TAG_INPUT / GITHUB_REF tag-check). The RELEASE-flag check below
  remains the single source of truth for release-mode gating, and
  the new composite action sets RELEASE for both formats from one
  place so the two checks can't diverge.

Verified locally:
- `nix develop --command task test-build-rpms` → green
- `nix develop --command task test-build-debs` → green
- workflow-setup-packaging.sh smoke tests pass on release-with-key
  (success), release-without-key (fatal as expected), and PR
  (success, no key, no error)
- actionlint clean on build-linux-packages.yml

Note for reviewer: branch protection rules referencing the old
status check names (build-rpm-packages / build-deb-packages) will
need to be updated to the new build-linux-packages name before
merge.

Addresses
- https://github.com/ava-labs/avalanchego/pull/5180/changes#r3306572758
- https://github.com/ava-labs/avalanchego/pull/5180/changes#r3306538214
Three locations in `.github/packaging/README.md` still named the
per-format workflows that (b254f0df89) deleted. Replace them with
references to the unified `build-linux-packages.yml` + the new
`build-package` composite action.

- "Main entrypoints" bullet block
- "CI behavior" intro sentence
- "Repository entrypoints" bullet block
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci This focuses on changes to the CI process devinfra

Projects

Status: In Progress 🏗️

4 participants